Skip to content

Index ONE_OF and NOT_ONE_OF lists when the configuration loads - #13486

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 17 commits into
masterfrom
leo.romanovsky/ffe-one-of-index
Nov 19, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 17 commits into
masterfrom
leo.romanovsky/ffe-one-of-index

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

ONE_OF and NOT_ONE_OF conditions use a hash index that the parser builds when the configuration loads. A membership test costs about 2–4 ns at every list size. Before this change, each evaluation scanned the whole list.

The results do not change. This includes the numeric leniency (42 matches "042") and the errors that the scan raises at an element that it cannot convert to a number.

Motivation

Customers target large lists of accounts, users, and devices with ONE_OF. The scan cost grows with the list size, and with a number attribute it also parses each element as a double on each evaluation:

  • 10,000-element list, string attribute: about 25 µs for each condition.
  • 10,000-element list, number attribute: about 90 µs and 600 KB of garbage for each condition.

Changes

  • ufc/MembershipIndex (new class):
    • strings: a flat open-addressing table of String.valueOf(element) for the elements that are not numbers, modeled on StringIndex in internal-api (which this module cannot depend on): String[] slots, cached int[] hashes that gate equals, at most half full. It keeps each string's first position only when the list has a number element, as only then does the position decide a match.
    • numeric and numbers: open-addressing tables with primitive long keys (the bits of the double value, with -0.0 stored as 0.0). They map each element that converts to a number, and each number element, to its first position. A lookup does not box. The slot is the high bits of a Fibonacci product, so whole numbers and powers of two spread over the table. A list without numeric strings uses one table for both, and a list without numbers or without strings shares one empty table.
    • lookup(attribute) returns MATCH, NO_MATCH, the element where the scan throws, or SCAN. A number attribute of any type is looked up by value, and any other attribute by its string.
  • UniversalFlagConfigParser: builds the index for each ONE_OF and NOT_ONE_OF list of 2 or more elements.
  • ConditionConfiguration: new transient field membership.
  • FlagEvaluator.isOneOf: uses the index. When the scan throws, it compares the attribute with the element where the scan throws, which throws the same exception. It runs the scan (scanOneOf, the code from before) when the index returns SCAN or when no index exists.
  • MembershipIndexTest (new): bounds the probe lengths of hits and misses in the number table for integers, negative integers, powers of two, prices in cent steps and ending in .99, large IDs, and integers mixed with powers of two in one list.
  • MembershipIndexStringTableTest (new): a collision chain that wraps past the end of the table, misses that collide with every key, the two strings whose hash code is 0, repeated keys, lists without numbers or without strings, mixed lists in list order, and the empty list.
  • UniversalFlagConfigParserTest: the parser builds the index for lists of 2 or more elements, and not for 1 element.
  • FlagEvaluatorMembershipTest (new):
    • A differential test: 4,000 random lists × 2 operators × 34 attributes (about 272,000 lookups). It requires the same value, reason, error code, and error message from the index and from the scan. The lists mix numeric strings, non-numeric strings, numbers, booleans, null, nested lists and maps, "-0", "NaN", "Infinity", " 42 ", and "4.2e1". The attributes are strings, integers, longs (one above 2^53), doubles (-0.0, NaN), booleans, an Instant, a list, and maps, one of them equal to a nested map that prints its keys in another order.
    • Table cases for the error ordering, a 10,000-element list, and the minimum indexed size, and a check that an error does not iterate the list.

Results

Membership structures alone, JMH AverageTime, ns per lookup (half of the keys are present):

Structure 2 8 32 1,000 10,000
String attribute: scan (before) 7.6 34.4 104 2,593 25,423
String attribute: TreeSet 5.0 9.2 15.7 31.0 57.3
String attribute: binary search on a sorted array 5.2 8.7 13.2 27.0 43.1
String attribute: HashMap 2.2 2.4 2.5 3.8 4.1
Number attribute: scan (before) 24.1 72.1 265 9,182 89,502
Number attribute: TreeMap 2.6 5.7 7.3 17.5 53.0
Number attribute: binary search on a sorted long[] 2.9 3.5 5.2 13.6 26.8
Number attribute: primitive hash table (this change) 1.8 2.0 2.0 2.1 2.2

This table measured a HashMap for strings. The string index is now a flat table, which the end-to-end runs below measure against the HashMap.

Scale suite results

Same machine and settings as the scale-suite baseline of #13485: AWS workspace, Intel Xeon Platinum 8175M, 16 vCPUs, JDK 21.0.10, JMH AverageTime, 1 fork, 3 warm-up and 5 measurement iterations of 2 s, -prof gc. Each row is the mean over the cases of the file. The runs were back to back.

Case file master This PR Change
scale/one-of (10,000-entry lists) 137 µs, 98,869 B 272 ns, 374 B 506× faster
scale/one-of-many (500 × 100 UUIDs, 1,000 × 256 chars, 50 × 200) 195 µs, 393 B 11.6 µs, 368 B 17× faster
scale/one-of-small-numbers ("1".."1000" and 53 powers of two) 11.8 µs, 15,424 B 307 ns, 380 B 38× faster
scale/rules, rules-sweep, time-windows, variations, flags 3.29 µs / 28.5 µs / 801 ns / 53.1 µs / 215 ns 3.55 µs / 31.3 µs / 842 ns / 53.9 µs / 231 ns within noise

A smoke run against master at 4075f1f (same machine, JDK and JMH settings, back to back, load average 3 to 13) gives: one-of 115 µs and 80,421 B on master against 372 ns ± 157 and 374 B on this PR; one-of-many 156 µs against 11.5 µs ± 4.4; one-of-small-numbers 11.5 µs and 15,424 B against 316 ns ± 76 and 380 B. The other case files stayed within the error. This is a shared workspace, so the errors on this PR's rows are wide; the ratios match the table.

one-of-many still costs 11.6 µs, because 500 allocations each do one lookup. The remaining cost is the allocation scan, not the lists.

The cost at load is in the next section.

Retained memory

The parsed configuration lives as long as the process, and every concurrent mark traces it, so the object count matters as much as the bytes. JOL GraphLayout.parseInstance on the parsed scale configuration (10,030 flags; 576 indexed lists of 92,093 values), JDK 21.0.10:

Retained Objects Allocated by one parse Parse time
master at 4075f1f 24.75 MB 805,330 90.5 MB 109 ms ± 19
String index as a HashMap 29.80 MB (+5.05 MB) 936,427 (+131,097) 99.0 MB (+8.5 MB) 110 ms ± 9
String index as a flat table (this PR) 27.45 MB (+2.69 MB) 807,650 (+2,320) 97.5 MB (+7.0 MB) 119 ms ± 16

Allocation and parse time come from ScaleSuiteLoadBenchmark with suite=scale and -prof gc, 3 forks of 5 × 2 s, at a load average of 6 to 8; the parse times are within the error of each other. The added allocation is the tables and the build arrays of each list, which the parse drops.

The HashMap held a node, and often a boxed Integer, for each value: 4.4 MB of the 5.05 MB. The flat table needs about 4 objects for each list. On the canonical fixture ufc-config.json, the index owns 20 objects instead of 43.

The flat table keeps the speed. ScaleSuiteBenchmark one-of rows, the two jars alternating, 6 forks of 5 × 2 s each, -prof gc, medians:

Case file master HashMap Flat table
scale/one-of 124 µs 278 ns 291 ns
scale/one-of-many 162 µs 9.24 µs 9.30 µs
scale/one-of-small-numbers 12.8 µs 276 ns 272 ns

Both allocate the same 374, 368, and 380 bytes per operation; master allocates 80,422, 393, and 15,424. Other sessions shared the workspace, with a load average between 2 and 9.

Decisions

  • A hash index, not a search tree or a binary search. The hash index is the fastest at every size in the table above. It is 14 times faster than a TreeSet and 10 times faster than a binary search at 10,000 elements. A JVM server can pay the build cost when the configuration loads.
  • Keep today's exact comparison rules. A plain string set would change the results for number attributes (42 would no longer match "042"), and the errors would be lost. The index stores the first position of each key and of the first element that cannot convert to a number, so it can tell which result the scan gives. When the scan's result is an error, the evaluator compares the attribute with the element where the scan throws, so the error type and message stay the same without a scan.
  • Primitive tables for numbers. A HashMap<Double, Integer> boxes each lookup. The long-keyed table does not allocate, and it is sized only for the elements that convert to numbers. A list of non-numeric IDs gets an almost empty number table.
  • The high bits of the hash pick the slot. The slot is the top bits of key * 0x9E3779B97F4A7C15L. For whole numbers below 2^21 and for powers of two, the low 32 bits of the key are zero, so the low bits of the product are too. A slot taken from them puts all of 1..1000 at one slot: 500 probes for each hit in MembershipIndexTest, against 1.06 with the high bits.
  • Index lists from 2 elements. Rechecked with the flat table: evaluating one ONE_OF condition over a 2-element string list costs 178 ns when the attribute is first (scan: 178 ns), 181 ns when it is last (scan: 198 ns), and 101 ns when it is absent (scan: 108 ns). At 16 elements, the last element costs 191 ns against 333 ns. A one-element list keeps the scan.
  • A flat string table, not a HashMap. The HashMap cost 60–80 bytes and one or two objects for each string element. The flat table costs 8 bytes for each slot (a reference and a hash), plus 4 for a position when the list has a number element, in 2 to 3 arrays for each list. Number elements cost 13 bytes for each slot. The configuration already holds the list and its strings, so the table only refers to them.

Additional Notes

Limitations:

  • One case still scans. An attribute that is not a string, a boolean, or a number, such as a List or a Map, against a list that has a nested list or object. Two equal maps can print their keys in a different order, so the string does not decide. This case costs O(n), as before.
  • Numbers still match leniently. 42 matches "042" in Java, as before.
  • The hashes are not seeded. A configuration could choose numbers or strings that share slots, which makes probes linear in the colliding keys. Configurations come from the Datadog backend.

Follow-ups:

  • Use the shared StringIndex. The string table can become StringIndex once the toolbox is shaded into this module (Shade the perf toolbox into dd-openfeature (FFE optimization) #13532).
  • Drop the original lists. A fully flat form (the values in one byte array, offsets, and an int[] hash table) could replace the lists themselves, taking the lists and their strings from about 8.5 MB to 4.7 MB on scale. The evaluator and the parser read the lists today, so this belongs in its own change.

Contributor Checklist

@leoromanovsky leoromanovsky added type: feature Enhancements and improvements tag: performance Performance related changes tag: ai generated Largely based on code generated by an AI or LLM comp: openfeature OpenFeature labels Nov 17, 2026
@datadog-prod-us1-5

datadog-prod-us1-5 Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 99.34%
• Overall Coverage: 60.68% (+0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 3edfb47 | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.07 s 14.03 s [-0.4%; +1.0%] (no difference)
startup:insecure-bank:tracing:Agent 12.96 s 13.03 s [-1.3%; +0.2%] (no difference)
startup:petclinic:appsec:Agent 17.14 s 17.06 s [-0.3%; +1.2%] (no difference)
startup:petclinic:iast:Agent 17.10 s 17.03 s [-0.4%; +1.2%] (no difference)
startup:petclinic:profiling:Agent 16.56 s 16.75 s [-2.2%; -0.1%] (maybe better)
startup:petclinic:sca:Agent 17.07 s 17.00 s [-0.5%; +1.3%] (no difference)
startup:petclinic:tracing:Agent 15.77 s 16.11 s [-6.2%; +2.0%] (no difference)

Commit: 3edfb472 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

leoromanovsky and others added 3 commits November 17, 2026 00:47
Add ScaleSuiteBenchmark, which evaluates each case of the opt-in scale suite of the shared FFE fixtures. The setup checks every expected value and reason first. Pin ffe-system-test-data to the scale suite PR head.
The scale benchmark sat on the JSON value change and checked JSON
integers by type, so it could not set a baseline on master. It now
compares JSON numbers by value, covers every case file of the scale and
scale-json suites, can isolate one case with the index parameter, and
prints each case's index. A load benchmark measures parsing each suite's
configuration and prints the heap it retains, so changes that move work
to load time show their cost.

The submodule points to the scale-suite fixture branch, which adds the
N x M dimensions and JSON values up to 10 MB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each ONE_OF or NOT_ONE_OF evaluation scanned its list and compared the
attribute with each element. With a number attribute, the scan also
parsed each element as a double. A 10,000-element list cost about 25 us
for a string attribute and 90 us for a number attribute.

The parser now builds a MembershipIndex for each list of 2 or more
elements: a HashMap for string comparisons and primitive long-keyed
tables for numeric comparisons, each mapping a key to the element's
first position. A lookup costs about 2-4 ns at every list size.

The index gives the same result as the scan, including the numeric
leniency (42 matches "042") and the errors that the scan raises at an
unconvertible element. When a result depends on such an error, or for
an attribute type that the index does not handle, the evaluator runs
the scan as before.
@leoromanovsky
leoromanovsky force-pushed the leo.romanovsky/ffe-one-of-index branch from db2555d to 2999aae Compare November 17, 2026 01:00
@leoromanovsky
leoromanovsky changed the base branch from master to leo.romanovsky/ffe-scale-suite-benchmark November 17, 2026 01:00
@leoromanovsky leoromanovsky added the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026 — with ddtool CLI
@leoromanovsky leoromanovsky removed the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 17, 2026
leoromanovsky and others added 2 commits November 17, 2026 03:58
…the index

The long table hashed with the low bits of a Fibonacci product. A whole
number below 2^21 has a key whose low 32 bits are zero, so small integers
and powers of two all started at slot 0, and a lookup in a list such as
1..1000 walked about 500 slots (792 ns instead of 5 ns). The slot now takes
the high bits of the product, which depend on every bit of the key.

- MembershipIndexTest: start slots of small integers, powers of two, prices
  in cent steps and large IDs; it fails on the old hash.
- UniversalFlagConfigParserTest: the parser builds the index for lists of two
  or more elements, and not for one element.
- FlagEvaluatorMembershipTest: attribute types that the index leaves to the
  scan (Instant, list, map), and nested list and object elements.
- A list without numeric strings shares one table for both number lookups.
- Comments match the code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scale benchmarks failed the forbidden API check on System.out, and
nothing outside a manual JMH run checked the scale cases, so a sibling
change that resolves a large configuration wrongly would go unnoticed.

- Add ScaleSuiteTest: evaluate every case of the scale and scale-json
  suites twice, and fail when the benchmark misses a case file. The
  cases, their check and the suite reader move to the test sources and
  the benchmarks reuse them; the reader uses JsonReading.
- Allow System.out in the benchmark setups (@SuppressForbidden).
- Check every case again after the measurement, fail on an error code,
  and reject expectations the check does not handle.
- Rerun the jmh task when the fixtures change.
- Measure retained heap after a first parse, as the smallest of three
  signed samples.
- Describe a row as the unweighted mean over the cases of a file, say
  that hooks are not measured and when to run three forks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leoromanovsky and others added 7 commits November 17, 2026 04:04
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixture PR now checks that the committed scale files match the
generator and fixes its docs. The case and configuration files are
unchanged, so the benchmark numbers do not change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixtures add a case file with ONE_OF lists of the integers 1 to
1000 and of the powers of two up to 2^52, which an index keyed on exact
doubles must still match. Pin them and benchmark the file; its 8 cases
pass on master.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The OpenFeature client passes the provider a new LayeredEvaluationContext
on each call, while the benchmark reused one MutableContext. A rule on
`id` when the context has no `id` reads the layered context's key set,
which it builds again on each call, so the benchmark did not show what
applications pay for it.

evaluateLayered wraps each case's context the way the client does, as
ExperimentationOptOutBenchmark.evaluateFreshLayeredContext does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leoromanovsky
leoromanovsky marked this pull request as ready for review November 19, 2026 05:27
@leoromanovsky
leoromanovsky requested review from a team as code owners November 19, 2026 05:27
@leoromanovsky
leoromanovsky requested review from bric3, dougqh, greghuels and typotter and removed request for a team and bric3 November 19, 2026 05:27
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T05:30:51.265755Z a114d9f Draft marked ready
🔒 Security Review ✅ Completed 2026-11-19T05:30:45.910524Z a114d9f Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a114d9fc59

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@datadog-prod-us1-5 datadog-prod-us1-5 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The index preserves numeric coercion and list-order error behavior, with scan fallback for compound comparisons. Tests and performance benchmarks were not rerun.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit a114d9f

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@leoromanovsky
leoromanovsky added this pull request to the merge queue Nov 19, 2026
@dd-octo-sts

dd-octo-sts Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-11-19 14:44:57 UTC ℹ️ Start processing command /merge


2026-11-19 14:45:01 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-11-19 14:56:46 UTC ⚠️ MergeQueue: This merge request build was cancelled

leo.romanovsky@datadoghq.com cancelled this merge request build

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

/merge -c

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-11-19 14:55:55 UTC ℹ️ Start processing command /merge -c

The HashMap of strings to first positions held a node and often a boxed
Integer per value. On the scale configuration it added 4.4 MB in about
130,000 long-lived objects that every concurrent mark traces.

A flat open-addressed table, modeled on StringIndex, keeps the strings,
their cached hashes and, only for lists with a number element, their
first positions. Lists without strings or without numbers share one
empty table.

Co-authored-by: Douglas Q Hawkins <douglas.hawkins@datadoghq.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copy link
Copy Markdown
Contributor Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-11-19 15:57:55 UTC ℹ️ Start processing command /merge


2026-11-19 15:58:02 UTC ℹ️ MergeQueue: Pull request is not mergeable yet

It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.

  • Run /code blockers to see what is blocking it.
  • Run /remove to cancel it.

2026-11-19 16:55:35 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-11-19 17:36:28 UTC ℹ️ MergeQueue: This merge request was merged

@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit ac1d3cf into master Nov 19, 2026
691 of 695 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the leo.romanovsky/ffe-one-of-index branch November 19, 2026 17:36
@github-actions github-actions Bot added this to the 1.70.0 milestone Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature tag: ai generated Largely based on code generated by an AI or LLM tag: performance Performance related changes type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants